Systems and methods for a dual modality sensor system

ABSTRACT

The present disclosure provides systems and methods for using two imaging modalities for imaging an object at two different resolutions. For example, the system may utilize a first modality (e.g., ultrasound or electromagnetic radiation) to generate image data at a first resolution. The system may then utilize the other modality to generate image data of portions of interest at a second resolution that is higher than the first resolution. In another embodiment, one imaging modality may be used to resolve an ambiguity, such as ghost images, in image data generated using another imaging modality.

If an Application Data Sheet (“ADS”) has been filed on the filing dateof this application, it is incorporated by reference herein. Anyapplications claimed on the ADS for priority under 35 U.S.C. §§119, 120,121, or 365(c), and any and all parent, grandparent, great-grandparent,etc., applications of such applications, are also incorporated byreference, including any priority claims made in those applications andany material incorporated by reference, to the extent such subjectmatter is not inconsistent herewith.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit of the earliest availableeffective filing date(s) from the following listed application(s) (the“Priority Applications”), if any, listed below (e.g., claims earliestavailable priority dates for other than provisional patent applicationsor claims benefits under 35 U.S.C. §119(e) for provisional patentapplications, for any and all parent, grandparent, great-grandparent,etc., applications of the Priority Application(s)).

PRIORITY APPLICATIONS

None.

RELATED APPLICATIONS

If the listings of applications provided herein are inconsistent withthe listings provided via an ADS, it is the intent of the Applicants toclaim priority to each application that appears in the PriorityApplications section of the ADS and to each application that appears inthe Priority Applications section of this application.

All subject matter of the Priority Applications and the RelatedApplications and of any and all parent, grandparent, great-grandparent,etc., applications of the Priority Applications and the RelatedApplications, including any priority claims, is incorporated herein byreference to the extent such subject matter is not inconsistentherewith.

This application is related to U.S. patent application Ser. No.14/203,401, filed on Mar. 10, 2014, titled SYSTEMS AND METHODS FORULTRASONIC POSITION AND MOTION DETECTION, and to U.S. patent applicationSer. No. 14/280,463, filed on May 16, 2014, titled SYSTEMS AND METHODSFOR ULTRASONIC VELOCITY AND ACCELERATION DETECTION, which applicationsare hereby incorporated by reference in its entirety.

TECHNICAL FIELD

This disclosure relates to systems and methods for generating an imageof an object within a region. Specifically, this disclosure providessystems and methods for using a dual modality sensor (e.g., ultrasoundand electromagnetic radiation) in combination with, for example,entertainment devices.

SUMMARY

A system may utilize a dual modality sensor to generate image data. Thesystem may use a first modality to generate coarse image data of anobject. The system may then use this coarse image data to identifyportions of interest of the object. The system may use a second modalityto generate fine image data of the identified portions of interest.

For example, in various embodiments, a system may include one or moreultrasonic transmitters and/or receivers to implement a first modality.In some embodiments the transmitter(s) and/or receiver(s) may beembodied as one or more transceivers. An ultrasonic transmitter may beconfigured to transmit ultrasound into a region bounded by one or moresurfaces. The ultrasonic receiver may receive direct ultrasonicreflections from one or more objects within the region. As described indetail below, the system may use the ultrasonic transmitters and/orreceivers to generate coarse image data of an object and identify, basedon the coarse image data, portions of interest of the object.

For instance, in certain examples, a system may also be configured toreceive, via an electromagnetic receiver, an electromagnetic reflectionfrom an object within a region. The system may generate fine image dataof identified portions of interest using the received electromagneticreflection. For example, after a portion of interest has been identifiedvia coarse image data, the system may receive electromagnetic radiationfrom the identified portion of interest and generate image data withgreater resolution than available in the coarse image data (referred toherein as fine image data).

In some embodiments, the dual modalities may be used to resolve at leastone ambiguity. For example, image data received from a first modalitymay include an ambiguity, such as a ghost image. In such an example, asecond modality may be utilized by the system to resolve the ambiguityintroduced by the first modality, e.g., the ghost image in the imagedata generated using the first modality. As a specific example, receivedelectromagnetic radiation can be utilized by a system to correct a ghostimage introduced by received ultrasound reflections.

Either of the two modalities discussed above (i.e., electromagneticimaging and ultrasound) can be utilized by the system to generate eitherthe fine image data or the coarse image data. For example, a firstembodiment may utilize ultrasound to generate coarse image data andelectromagnetic imaging to generate fine image data, whereas a secondembodiment may utilize electromagnetic imaging to generate coarse imagedata and ultrasound to generate fine image data.

The foregoing summary is illustrative only and is not intended to belimiting in any way. In addition to the illustrative aspects,embodiments, and features described above, further aspects, embodiments,and features are demonstrated with reference to the drawings and thefollowing detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a positioning system transmitting ultrasound towardthree persons within a bounded region.

FIG. 1B illustrates a direct ultrasonic reflection received by thepositioning system and the resulting “image” generated by thepositioning system.

FIG. 2A illustrates a positioning system rebounding ultrasound off thewall and then toward the three persons.

FIG. 2B illustrates a side view of the positioning system rebounding theultrasound off the wall and then toward the three persons.

FIG. 3A illustrates a plurality of ultrasonic reflectors configured tofacilitate the transmission, reflection, and/or reception of reboundedultrasound by the positioning system.

FIG. 3B illustrates a plurality of active ultrasonic reflectorsconfigured to facilitate the transmission, reflection, and/or receptionof rebounded ultrasound by the positioning system.

FIG. 4A illustrates an actively controlled ultrasonic reflector in afirst position configured to pivot with respect to the wall on which itis mounted to facilitate the transmission, reflection, and/or receptionof rebounded ultrasound by the positioning system.

FIG. 4B illustrates the actively controlled ultrasonic reflector in asecond position.

FIG. 5 illustrates a block diagram of a positioning system, according toone embodiment.

FIG. 6 illustrates a flow chart of a method for generating positionaldata describing a relative position of one or more objects within aregion.

FIG. 7A illustrates an ultrasonic system transmitting and receivingreflected ultrasound from a stationary object.

FIG. 7B illustrates an ultrasonic system transmitting ultrasound at afirst frequency and receiving reflected ultrasound at a second frequencyfrom an object moving away from the ultrasound system.

FIG. 7C illustrates an ultrasound system transmitting ultrasound at afirst frequency and receiving reflected ultrasound at a second frequencyfrom an object moving toward the ultrasound system.

FIG. 7D illustrates an ultrasonic system transmitting and receivingreflected ultrasound from a stationary object, similar to FIG. 7A.

FIG. 7E illustrates a timing delay in a reflected ultrasound from theobject as it moves away from the ultrasound system.

FIG. 8 illustrates ultrasound rebounded off of a reflector prior tobeing reflected by an object moving away from an ultrasound receiver.

FIG. 9. Illustrates an electromagnetic position detection system used inconjunction with an ultrasound velocity and/or acceleration detectionsystem.

FIG. 10 illustrates ultrasound reflected and/or rebounded from one ormore auxiliary reflectors.

FIG. 11 illustrates a plurality of ultrasonic systems for determiningvelocity and/or acceleration information from multiple directions.

FIG. 12 illustrates a method for determining velocity and/oracceleration information associated with a moving object.

FIG. 13A illustrates a dual modality system transmitting and receivingultrasound reflected off a person standing within a bounded region.

FIG. 13B illustrates a dual modality system generating coarse image dataof an object based on received ultrasonic reflections.

FIG. 13C illustrates a dual modality system identifying several portionsof interest on an object based on coarse image data generated fromultrasonic reflections.

FIG. 14 illustrates a dual modality system using electromagneticreflections in conjunction with ultrasound to receive additional imageinformation from identified portion of interests.

FIG. 15 illustrates a dual modality system generating fine image data ofportions of interest on a person.

FIG. 16 illustrates a method for generating an image using ultrasoundand electromagnetic radiation.

FIG. 17 illustrates another method for generating an image usingultrasound and electromagnetic radiation.

FIG. 18 illustrates a method for resolving ambiguities in an image usingultrasound and electromagnetic radiation.

DETAILED DESCRIPTION

A system may utilize a dual modality sensor system to generate imagedata. For instance, a system may utilize two imaging modalities forimaging an object at two different resolutions. That is, the system mayutilize a first modality (e.g., either ultrasound or electromagneticradiation) to generate image data of an object at a first resolution.The system may then utilize the other modality to generate image data ofportions of interest on the object (not necessarily the entire object)at a second resolution, where the second resolution is higher than thefirst resolution. Accordingly, the dual modalities may be used togenerate a coarse (i.e., lower resolution) image of the entire objectusing a first modality, identify portions of interest on the object, andthen generate a fine (i.e., higher resolution) image of the portions ofinterest using a second modality.

As a specific example, a system may transmit ultrasound, via a firstultrasonic transmitter, into the region. An ultrasonic receiver mayreceive ultrasonic reflections of the transmitted ultrasound from aplurality of sites on the object within the region. A processor maygenerate coarse image data of the object at a first resolution based onthe received ultrasonic reflections. The system may then identify aportion of interest on the object based on the coarse image data.Electromagnetic radiation may be received from the identified portion ofinterest on the object. Fine image data of the portion of interest onthe object may be generated at a second resolution based on the receivedelectromagnetic radiation. The second resolution may be greater than thefirst resolution. In some embodiments, the first resolution may begreater than the second resolution.

In some embodiments, a kinematic value may be determined that isassociated with the portion of interest on the object based on at leastone of the received electromagnetic radiation and the receivedultrasonic reflections. Similarly, in some embodiments, the state of anentertainment device may be modified based on the determined kinematicvalue associated with the portion of interest on the object.

In some embodiments, the coarse image data described above may begenerated based on the received electromagnetic reflections and the fineimage data may be generated based on the received ultrasonicreflections. In any of the various embodiments described herein, thereceived electromagnetic radiation may be generated by the system, byanother system, by an auxiliary electromagnetic radiation source, and/orcomprise ambient electromagnetic radiation, such as light.

In some embodiments, one imaging modality may be used to resolve anambiguity, such as ghost images, in image data generated using anotherimaging modality. For example, an image generated using ultrasoundimaging technologies may have an ghosting image ambiguity that can beresolved using an electromagnetic imaging technology (or even just anelectromagnetic position/distance detection technology).

For example, a system may include one or more ultrasonic transmittersand/or receivers, as well as one or more electromagnetic transmittersand/or receivers. Each of these different modalities may capture imagedata at different resolutions. The system may generate non- orless-important image data at a lower resolution while capturingimportant image data at a higher resolution. In some embodiments,generating only a portion of the image at a higher resolution may allowthe system to process the image data faster and in a more compressedmanner while still providing high resolution of portions of interest.

In some embodiments, the transmitter(s) and/or receiver(s) may beembodied as one or more transceivers. The ultrasonic transmitter(s)and/or receiver(s) may be operated by the system concurrently with theelectromagnetic receiver(s) or in sequential order before or after theelectromagnetic receiver(s). The ultrasonic transmitter(s) andreceiver(s) may be used in combination with the electromagnetic receiverto generate image data.

One or more of the electromagnetic and/or ultrasonic transmitters,receivers, and/or transceivers may comprise a piezoelectric transducerthat may be part of a single transducer system or an array oftransducers. In some embodiments, the transducers may comprise or bemade from metamaterials. A flat sub-wavelength array of ultrasonictransducers may be used in conjunction with the embodiments describedherein, such as those utilizing arrays of metamaterials.

The dual modality sensor system may be configured to utilize differentfrequency spectrums. An ultrasonic transmitter used on such a system maybe configured to transmit ultrasound into a region bounded by one ormore surfaces. The ultrasound may be between 20 kHz and 250 kHz. In oneembodiment, the ultrasound is specifically between 35 kHz and 45 kHz. Anelectromagnetic transmitter and/or receiver may also be used on such asystem to transmit and/or receive a range of electromagnetic radiationfrequencies. For example, a system may be configured to useelectromagnetic microwave, terahertz, infrared, visible, and/orultraviolet radiation. A dual modality sensor system may use the twomodalities to produce more detailed image data and/or to correctambiguities introduced by one of the modalities.

For example, the system may use a first modality to generate coarseimage data of an object, and, to get more detailed data about anidentified portion of interest, the system may use a second modality togenerate fine image data of the identified portion of the object. Forexample, if the first modality produces an image with a low resolutionof an object, the second modality may be used to provide a higherresolution of portions of interest on the object. Another embodiment mayinclude a first modality that introduces an ambiguity into the imagedata. To correct the ambiguity, the system may use a second modalitythat isn't susceptible to the same type of ambiguity.

For example, the system may include one or more ultrasonic transmittersand/or receivers. The system may use the ultrasonic receivers and/ortransmitters to generate coarse image data. For instance, an ultrasonictransmitter may be configured to transmit ultrasound into a region. Anultrasonic receiver may receive ultrasonic reflections from one or moreobjects within the region. Based on these received ultrasonicreflections, the system may generate coarse image data of the one ormore objects via a processor.

The system may identify portions of interest on the object using thecoarse image data. For example, the system may identify a person's hand,finger, arm, leg foot, toe, torso, neck, head, mouth, lip, and/or eye.The portion of interest identified may be based on a state of anentertainment device. Once one or more portions of interest have beenidentified, the system may use a second modality (e.g., electromagneticradiation) to gather further details about the portion of interest.

For instance, the system may also electromagnetic receiver(s) and/ortransmitter(s). The system may generate fine image data of theidentified portions of interest using received electromagneticreflections. For example, after a portion of interest has beenidentified using the coarse image data, the system may receiveelectromagnetic radiation from the identified portion of interest andgenerate higher resolution image of the identified portions of interest.

In some embodiments, a second modality may be used to resolve at leastone ambiguity inherent in or caused by the usage of the first modality.For example, image data generated using the first modality may includean ambiguity. For example, image data generated via ultrasound may haveghost images inherent in the image data. In such an example, a secondmodality (e.g., electromagnetic radiation) may be utilized by the systemto resolve the ambiguity introduced by the first modality. For instance,received electromagnetic radiation can be utilized by the system toremove ghost images in the image data generated using the ultrasoundreflections.

Either of the two modalities discussed above (i.e., electromagneticimaging and ultrasound) may be utilized by the system to generate eitherfine image data or coarse image data. For example, a first embodimentmay utilize ultrasound to generate coarse image data and electromagneticimaging to generate fine image data, whereas a second embodiment mayutilize electromagnetic imaging to generate coarse image data andultrasound to generate fine image data.

A kinematic value associated with the object or a specific portion ofinterest on the object may be determined. The kinematic value of anobject may comprise the position, velocity, and/or acceleration of theobject. The kinematic values may be based on the receivedelectromagnetic radiation and/or the received ultrasonic reflections.

In some embodiments, the direct ultrasound may be reflected from a firstportion of an object and the rebounded ultrasound may be reflected froma second, different portion of the object. Positional data may bedetermined using the received ultrasonic reflections. Direct positionaldata may correspond to a first directional component of the position ofthe object and the rebounded positional data may correspond to a seconddirectional component of the position of the object. Similarly, one ormore direct and/or rebounded ultrasonic reflections may be used todetermine velocity and/or acceleration. For example, velocity and/oracceleration information may be determined using a Doppler shift thatcorresponds to a motion of the reflecting object.

In some embodiments, received ultrasonic reflections (direct orrebounded) may be used to determine positional data. Positional datasampled at various times may be used to determine and/or estimatecurrent and/or future velocity and/or acceleration informationassociated with an object. In other embodiments, as described herein,velocity and/or acceleration information may be calculated based on adetected shift in ultrasound reflected by an object. For example, asystem may detect a Doppler shift in ultrasound reflected by an objectrelative to the transmitted ultrasound. A shift to a longer wavelengthmay indicate that the object is moving away from the ultrasonicreceiver. A shift to a shorter wavelength may indicate that the objectis moving toward the ultrasonic receiver. The detected shift may berelated to a frequency shift, a wavelength shift, a phase shift, atime-shifted reflection, and/or other ultrasonic shift.

Any number of direct and/or rebounded ultrasonic reflections may beobtained from one or more objects within a region to obtain velocityand/or acceleration data over a period of time and/or to obtain moreaccurate velocity and/or acceleration data with multiple data points.The transmitted ultrasound may be transmitted as directional ornon-directional ultrasonic pulses, continuously, in a modulated(frequency, amplitude, phase, etc.) fashion, and/or other format. Theultrasonic transmissions may be spaced at regular intervals, on demand,and/or based on the reception of a previously transmitted ultrasonictransmission. Direct and rebounded ultrasound pulses may be transmittedat the same time, or either one can be transmitted before the other.

Rebounded ultrasonic reflections may be defined as ultrasonicreflections that, in any order, reflect off at least one surface inaddition to the object. For example, the rebounded ultrasonicreflections may be reflected off any number of surfaces and/or objects(in any order) prior to being received by the ultrasonic receiver.

A mapping or positioning system may generate positional data associatedwith one or more of the object(s) based on the direct ultrasonicreflection(s) and/or the rebounded ultrasonic reflection(s). Thepositional data may comprise a centroid of the objects, atwo-dimensional mapping of the object, an image of the object, afalse-color representation of the object, an information representation(blocks, squares, shadows, etc.) of the object, a three-dimensionalmapping of the object, one or more features of the object, and/or otherinformation.

The velocity and/or acceleration data may be defined with respect to oneor more surfaces of the region, the ultrasonic velocity/accelerationsystem, a receiver of the system, and/or a transmitter of the system.The one or more objects within the region may comprise machinery,robots, furniture, household property, people in general, gamers, humancontrollers of electronic devices, electronic devices, fixtures, and/orother human or non-human objects.

The object may comprise a specific portion of a person, such as a hand,finger, arm, leg, foot, toe, torso, neck, head, mouth, lip, or eye. Insome embodiments, rebounded ultrasonic transmissions may be reflectedoff an ultrasonic reflector disposed within the room. In someembodiments, the ultrasonic reflectors may be mounted and/or otherwisepositioned within the region. In other embodiments, the ultrasonicreflectors may be held, worn, and/or otherwise in the position of theuser or operator of the ultrasonic positioning system. The ultrasonicreflectors may modify a characteristic of the reflected ultrasound,facilitating the identification of the received rebounded ultrasonicreflections.

Ultrasonic reflectors may comprise passive, active, and/or activelymoved/pivoted ultrasonic reflectors for controlling the direction inwhich ultrasound rebounds and/or otherwise travels within the region.For example, the ultrasonic reflector may be configured to modify one ormore of the frequency, phase, and/or amplitude of the reboundedultrasound. The modified characteristic may facilitate thedifferentiation of the direct ultrasonic reflections and the reboundedultrasonic reflections. In some embodiments the direct and reboundedsignals can be differentiated using knowledge of the transmission orreception directions of the respective beams. In some embodiments, thedirect and rebounded signals can be differentiated using knowledge ofthe time-of-flight of the respective beams. In some embodiments, thedirection of a reflected beam (and hence directional characteristics ofits delivered positional information) can be determined by knowledge ofthe orientation of the reflecting surface and its reflectivecharacteristics. For example, ultrasonic reflection from a surface maybe dominated by specular reflection, thereby allowing straightforwarddetermination of the rebound geometry.

The mapping or positioning system may also generate velocity and/oracceleration data using the rebounded ultrasonic reflection of theobject(s) from the one or more surfaces. It will be appreciated that arebounded ultrasonic reflection from a surface may be rebounded off thesurface first and then the object, or off the object first and then thesurface.

The system may then generate enhanced velocity and/or acceleration databy combining the direct velocity and/or acceleration data and therebounded velocity and/or acceleration data. The enhanced velocityand/or acceleration data may be a concatenation of the direct andrebounded velocity and/or acceleration data or a simple or complexfunction of the direct and rebounded velocity and/or acceleration data.

For example, in one embodiment, the direct and rebounded velocity and/oracceleration data may comprise only time-of-flight information, which,based upon air sound-speed, can be converted to transit distanceinformation for each beam. In such embodiments, the direct velocityand/or acceleration data provides a range from the transceiver to theobject, i.e., leaving the velocity and/or acceleration undefined along atwo-dimensional spherical surface. Each potential object position alongthis spherical surface leads, e.g., assuming specular reflections, to adistinct time-of-flight for the rebounded beam from one surface (wall,ceiling, floor); this restricts the locus of possible velocities and/oraccelerations of the object to a one-dimensional arc along the sphericalsurface, thereby improving the velocity and/or acceleration estimate(s).

The system can further refine the velocity and/or acceleration data byanalyzing rebound data from a second surface. In the current example,each potential object position along the spherical surface (obtained bythe time-of-flight of the direct beam) defines a first time-of-flightfor ultrasound rebounded from the first surface and a secondtime-of-flight for ultrasound rebounded from the second surface;knowledge of both times-of-flight determines the object's position. Itis clear that time-of-flight data from other surfaces can, by “overdefining” the problem, improve the positional estimate, e.g., byreducing sensitivity to measurement errors, to the effects of diffusereflections, etc. In other embodiments, the direct and reboundedvelocity and/or acceleration data may comprise directional information.

For example, directional information for direct ultrasound can identifythat the object (or a specified portion of it) lies along a known ray,thereby providing two components of its velocity and/or acceleration.Information from rebounded ultrasound can then provide additionalacceleration and/or velocity data sufficient to identify the thirdcomponent of the object's velocity and/or acceleration, i.e., along theray. The rebounded ultrasound may provide time-of-flight information;each object velocity and/or acceleration along the ray corresponds to adifferent time-of-flight for rebounded ultrasound from a surface, so themeasured time-of-flight identifies the object's location, velocity,and/or acceleration. The rebounded ultrasound may provide directionalinformation (for either transmission or reception); the intersection ofthis rebound ray with the direct ray serves to identify the object'slocation, velocity, and/or acceleration.

The enhanced velocity and/or acceleration data may be further enhancedor augmented using additional velocity and/or acceleration data obtainedvia direct or rebounded ultrasonic reflections and/or other velocityand/or acceleration data, such as velocity and/or acceleration dataobtained via other means/systems/methods (e.g., laser detection,cameras, etc.). The direct and the rebounded velocity and/oracceleration data may provide velocity and/or acceleration data for theobject at the same or different times, depending on the time at whichthey are reflected from the object. The enhanced positional data may beanalyzed using a dynamical model, e.g., a Kalman filter, designed tocombine velocity and/or acceleration data corresponding to differenttimes or directional components, using them together with, and toimprove, estimates of the object's present and/or future motion.

In some embodiments, direct ultrasonic reflections may not be used.Rather, a first rebounded ultrasonic reflection and a second reboundedultrasonic reflection may be used to generate velocity and/oracceleration data. It is appreciated that any number of direct orrebounded ultrasonic reflections may be used to identify a position,velocity, acceleration, and/or other movement information of an objectwithin a region. In various embodiments, the velocity and/oracceleration data gathered using ultrasonic reflections may be combinedwith other velocity and/or acceleration data, such as infrared, velocityand/or acceleration data provided by manual input, echo location, sonartechniques, laser, and/or the like.

In various embodiments, one or more local, remote, or distributedsystems and/or system components may transmit ultrasound via anultrasonic transmitter into a region. The received ultrasound mayinclude both direct reflections and rebounded reflections. Velocityand/or acceleration data from both the direct reflections and therebounded reflections may be used to obtain velocity and/or accelerationdata that more accurately and/or more quickly describes the relativevelocity and/or acceleration data of one or more objects within theregion.

Embodiments may include various steps, which may be embodied inmachine-executable instructions to be executed by a computer system. Acomputer system includes one or more general-purpose or special-purposecomputers (or other electronic devices). The computer system may includehardware components that include specific logic for performing the stepsor may include a combination of hardware, software, and/or firmware.

Embodiments may also be provided as a computer program product includinga computer-readable medium having stored thereon instructions that maybe used to program a computer system or other electronic device toperform the processes described herein. The computer-readable medium mayinclude, but is not limited to: hard drives, floppy diskettes, opticaldisks, CD-ROMs, DVD-ROMs, ROMs, RAMs, EPROMs, EEPROMs, magnetic oroptical cards, solid-state memory devices, or other types ofmedia/computer-readable media suitable for storing electronicinstructions.

Computer systems and the computers in a computer system may be connectedvia a network. Suitable networks for configuration and/or use asdescribed herein include one or more local area networks, wide areanetworks, metropolitan area networks, and/or Internet or IP networks,such as the World Wide Web, a private Internet, a secure Internet, avalue-added network, a virtual private network, an extranet, anintranet, or even standalone machines which communicate with othermachines by physical transport of media. In particular, a suitablenetwork may be formed from parts or entireties of two or more othernetworks, including networks using disparate hardware and networkcommunication technologies.

One suitable network includes a server and several clients; othersuitable networks may contain other combinations of servers, clients,and/or peer-to-peer nodes, and a given computer system may function bothas a client and as a server. Each network includes at least twocomputers or computer systems, such as the server and/or clients. Acomputer system may include a workstation, laptop computer,disconnectable mobile computer, server, mainframe, cluster, so-called“network computer” or “thin client,” tablet, smart phone, personaldigital assistant or other hand-held computing device, “smart” consumerelectronics device or appliance, medical device, or a combinationthereof.

The network may include communications or networking software, such asthe software available from Novell, Microsoft, Artisoft, and othervendors, and may operate using TCP/IP, SPX, IPX, and other protocolsover twisted pair, coaxial, or optical fiber cables, telephone lines,radio waves, satellites, microwave relays, modulated AC power lines,physical media transfer, and/or other data transmission “wires” known tothose of skill in the art. The network may encompass smaller networksand/or be connectable to other networks through a gateway or similarmechanism.

Each computer system includes at least a processor and a memory;computer systems may also include various input devices and/or outputdevices. The processor may include a general purpose device, such as anIntel®, AMD®, or other “off-the-shelf” microprocessor. The processor mayinclude a special purpose processing device, such as an ASIC, SoC, SiP,FPGA, PAL, PLA, FPLA, PLD, or other customized or programmable device.The memory may include static RAM, dynamic RAM, flash memory, one ormore flip-flops, ROM, CD-ROM, disk, tape, magnetic, optical, or othercomputer storage medium. The input device(s) may include a keyboard,mouse, touch screen, light pen, tablet, microphone, sensor, or otherhardware with accompanying firmware and/or software. The outputdevice(s) may include a monitor or other display, printer, speech ortext synthesizer, switch, signal line, or other hardware withaccompanying firmware and/or software.

The computer systems may be capable of using a floppy drive, tape drive,optical drive, magneto-optical drive, or other means to read a storagemedium. A suitable storage medium includes a magnetic, optical, or othercomputer-readable storage device having a specific physicalconfiguration. Suitable storage devices include floppy disks, harddisks, tape, CD-ROMs, DVDs, PROMs, RAM, flash memory, and other computersystem storage devices. The physical configuration represents data andinstructions which cause the computer system to operate in a specificand predefined manner as described herein.

Suitable software to assist in implementing the invention is readilyprovided by those of skill in the pertinent art(s) using the teachingspresented here and programming languages and tools, such as Java,Pascal, C++, C, database languages, APIs, SDKs, assembly, firmware,microcode, and/or other languages and tools. Suitable signal formats maybe embodied in analog or digital form, with or without error detectionand/or correction bits, packet headers, network addresses in a specificformat, and/or other supporting data readily provided by those of skillin the pertinent art(s).

Several aspects of the embodiments described will be illustrated assoftware modules or components. As used herein, a software module orcomponent may include any type of computer instruction or computerexecutable code located within a memory device. A software module may,for instance, include one or more physical or logical blocks of computerinstructions, which may be organized as a routine, program, object,component, data structure, etc., that performs one or more tasks orimplements particular abstract data types.

In certain embodiments, a particular software module may includedisparate instructions stored in different locations of a memory device,different memory devices, or different computers, which togetherimplement the described functionality of the module. Indeed, a modulemay include a single instruction or many instructions, and may bedistributed over several different code segments, among differentprograms, and across several memory devices. Some embodiments may bepracticed in a distributed computing environment where tasks areperformed by a remote processing device linked through a communicationsnetwork. In a distributed computing environment, software modules may belocated in local and/or remote memory storage devices. In addition, databeing tied or rendered together in a database record may be resident inthe same memory device, or across several memory devices, and may belinked together in fields of a record in a database across a network.

Much of the infrastructure that can be used according to the presentinvention is already available, such as: general purpose computers,computer programming tools and techniques, computer networks andnetworking technologies, digital storage media, authentication, accesscontrol, and other security tools and techniques provided by publickeys, encryption, firewalls, and/or other means.

The embodiments of the disclosure are described below with reference tothe drawings, wherein like parts are designated by like numeralsthroughout. The components of the disclosed embodiments, as generallydescribed and illustrated in the figures herein, could be arranged anddesigned in a wide variety of different configurations. Furthermore, thefeatures, structures, and operations associated with one embodiment maybe applicable to or combined with the features, structures, oroperations described in conjunction with another embodiment. In otherinstances, well-known structures, materials, or operations are not shownor described in detail to avoid obscuring aspects of this disclosure.

Thus, the following detailed description of the embodiments of thesystems and methods of the disclosure is not intended to limit the scopeof the disclosure, as claimed, but is merely representative of possibleembodiments. In addition, the steps of a method do not necessarily needto be executed in any specific order, or even sequentially, nor do thesteps need to be executed only once.

FIG. 1A illustrates a dual modality positioning system 110 transmittingultrasound 120 toward three persons 151, 152, and 153 in a group 150within a bounded region 100. As illustrated, the bounded region 100 isbounded by a floor 141, a left wall 142, a back wall 143, a right wall144, and a ceiling 145. A front wall (not shown), may also bound theregion 100.

The dual modality positioning system 110 may transmit the ultrasound 120as directional ultrasonic pulses, continuously, in a modulated fashion(frequency, amplitude, phase, etc.), and/or in another format. Theultrasound 120 may be transmitted directly toward the persons 151, 152,and 153. The ultrasound 120 may be transmitted indirectly toward thepersons 151, 152, and 153.

In various embodiments, the dual modality positioning system 110 may beany shape or size and/or may comprise a plurality of distributedcomponents. The illustrated embodiment is merely an example and is notintended to convey any information regarding shape, size, configuration,or functionality. In various embodiments, the dual modality positioningsystem 110 may include an array of transducers, such as piezoelectrictransducers, configured to transmit and/or receive ultrasound and/orelectromagnetic radiation. The dual modality positioning system 110 maybe configured with a first plurality of transducers 112 (or a singletransducer) for transmitting ultrasound and/or electromagnetic radiationand a second plurality of transducers 113 (or a single transducer) forreceiving ultrasound.

FIG. 1B illustrates a direct ultrasonic reflection 121 received by thedual modality positioning system 110. As illustrated, the directultrasonic reflection 121 may convey coarse image information in arelatively two-dimensional fashion in which the three persons 151, 152,and 153 are viewed as a single object 160, or as three distinct objects(161, 162, and 163) in substantially the same plane. FIG. 1B illustratesa visual representation of the received direct reflection of ultrasound121. The actual positional data received may be at a higher or lowerresolution depending on the sampling rates, accuracy, processing bitdepth, frequency(ies) of ultrasound used, etc.

FIG. 2A illustrates a dual modality positioning system 210, similar tothat described in conjunction with FIGS. 1A and 1B, in which ultrasound225 is transmitted toward a surface bounding the region 200. In theillustrated embodiment, the rebounding surface is left wall 242. It isappreciated that ultrasound may be rebounded off one or more of leftwall 242, floor 241, back wall 243, right wall 244, and/or ceiling 245.Similarly, electromagnetic radiation may be rebounded off one or more ofleft wall 242, floor 241, back wall 243, right wall 244, and/or ceiling245. In some embodiments, the system may receive electromagneticradiation from other sources (e.g., ambient light in the region).

As used herein, the terms rebound and rebounding may include any type ofreflection, refraction, and/or repeating that may or may not include aphase, frequency, modulation, and/or amplitude change. Rebounding may beperformed by the outer surface of the surface, an inner portion of thesurface, or an object disposed on, in, or behind the surface (e.g.,exterior paint, drywall, internal metal, studs, interior coatings,mounted panels, etc.).

The ultrasound may ultimately be rebounded 227 to reflect off persons251, 252, and 253 at a different angle than that obtained in FIGS. 1Aand 1B. The illustrated embodiment shows the rebounded ultrasound 227reflecting off the left wall 242 prior to the persons 251-253. However,the ultrasound may reflect off persons 251-253 prior to the left wall242 instead. Ultimately, ultrasound 225 may be rebounded and/orreflected by persons 251-253 and one or more of surfaces/walls 241-245in any order and then be received by positioning system 210.

FIG. 2B illustrates a side view of the dual modality positioning system210 described in conjunction with FIG. 2A with the rebounded ultrasound226 being received after reflecting off persons 251-253, at location228, and rebounding off left wall 242. FIG. 2B also shows a front wall246. In some embodiments, all of the ultrasound may be transmittedagainst a front wall 246 to more evenly distribute ultrasound throughoutthe region (i.e., a wider effective beam width).

As illustrated in FIG. 2B, the positional data obtained by the reboundedultrasound 226 may provide coarse image information not available viathe direct reflections shown in FIGS. 1A and 1B, e.g., due to one objectpreventing direct ultrasound from reaching a second object (or anotherportion of the first object). For instance, the visual representation ofthe positional data obtained illustrates three distinct objects 261,262, and 263 that are clearly in distinct planes relative to the dualmodality positioning system 210. For instance, the positional datagenerated based on the rebounded ultrasound in FIG. 2B shows a distanceD between object 262 and objects 261 and 263. Such a distance D may bedifficult to determine or determined differently if only directreflections were available (as in FIGS. 1A and 1B).

FIG. 3A illustrates a plurality of ultrasonic and/or electromagneticradiation reflectors 371, 372, 373, and 374 secured to, mounted to,positioned within, and/or integrally formed with one or more of thesurfaces 341, 342, 343, 345, and 346. In some embodiments, auser/subject may hold or otherwise control a portable ultrasonic and/orelectromagnetic radiation reflector 375. The ultrasonic reflectors371-375 may facilitate the transmission, reflection, and/or reception ofrebounded ultrasound by the dual modality positioning system 310.

The ultrasonic and/or electromagnetic radiation reflectors may comprisepassive, active, and/or actively moved/pivoted ultrasonic reflectors forcontrolling the direction in which ultrasound rebounds and/or otherwisetravels within the region. For example, the ultrasonic and/orelectromagnetic radiation reflector may be configured to modify one ormore of the frequency, phase, and/or amplitude of the reboundedultrasound and/or electromagnetic radiation. The modified characteristicmay facilitate the differentiation of the direct ultrasonic and/orelectromagnetic radiation reflections and the rebounded ultrasonicand/or electromagnetic radiation reflections.

The dual modality mapping or positing system 310 may generate positionaldata associated with one or more of the object(s) based on the directultrasonic and/or electromagnetic radiation reflection(s) (e.g., FIGS.1A and 1B) and/or the rebounded ultrasonic and/or electromagneticradiation reflection(s) (e.g., FIGS. 2A and 2B). The positional data maycomprise a centroid of the objects, a two-dimensional mapping of theobject, an image of the object, a false-color representation of theobject, an information representation (blocks, squares, shadows, etc.)of the object, a three-dimensional mapping of the object, one or morefeatures of the object, and/or other information. The positional datagenerated via one modality (i.e., the ultrasonic or the electromagneticradiation) may be at a higher or lower resolution that the positionaldata generated by the other modality.

The positional data may be defined with respect to one or more surfacesof the region, the dual modality positioning system 310, a receiver ofthe positioning system 312, and/or a transmitter 313 of the positioningsystem. The one or more objects within the region may comprisemachinery, robots, furniture, household property, people in general,garners, human controllers of electronic devices, electronic devices,fixtures, and/or other human or non-human objects.

The object may comprise a specific portion of a person, such as a hand,finger, arm, leg, foot, toe, torso, neck, head, mouth, lip, and/or eye.As illustrated in FIGS. 3A and 3B, rebounded ultrasonic transmissionsmay be reflected off an ultrasonic reflector 371-375 disposed within theroom. In some embodiments, the ultrasonic reflectors may modify acharacteristic of the reflected ultrasound, facilitating theidentification of the received rounded ultrasonic reflections.

FIG. 3B illustrates a plurality of active ultrasonic reflectors 391-394configured to facilitate the transmission, reflection, and/or receptionof rebounded ultrasound by the positioning system. As illustrated,active ultrasonic reflectors 391-394 may be connected to a power source,such as batteries, solar cells, heat converts, outlets 380, and/or othersuitable power source. In some embodiments, the ultrasound itself mayprovide the power source.

FIG. 4A illustrates an actively controlled ultrasonic reflector 472 in afirst position. A dual modality positioning system 410 may be incommunication with the ultrasonic reflector 472, or the ultrasonicreflector 472 may be autonomous. In various embodiments, the positioningsystem 410 may transmit ultrasound 425 toward the persons 451, 452, and453 or toward the wall 442, as illustrated. The ultrasound 425 may thenbe rebounded off the wall 442 or reflected by the persons 451-453,respectively.

FIG. 4B illustrates the actively controlled ultrasonic reflector 472 ina second position. The ultrasonic reflector 472 may be pivoted and/orcontrolled by a pivot control 495.

In some embodiments, pivot control 495 may change other reflective,absorptive, and/or refractive properties of the ultrasonic reflector472, in addition to its direction. For example, an ultrasonic reflector472 may have specific ultrasonic or other acoustic absorptiveproperties. A pivot control 495 may adjust the pivoting and/or acousticand/or electrical properties.

FIG. 5 illustrates a block diagram of a positioning system 500,according to one embodiment. As illustrated, a positioning system 500may include a processor 530, a memory 540, and possibly a network 550 orother data transfer interface. A bus 520 may interconnect variousintegrated and/or discrete components. Various modules may beimplemented in hardware, software, firmware, and/or a combinationthereof.

An ultrasonic transmitter module 580 may be configured to transmitultrasound in any of the various forms and/or methods described herein.An ultrasonic receiver module 582 may be configured to receive a directultrasonic reflection from an object within a region. Additionally, theultrasonic receiver module 582 may be configured to receive reboundedultrasonic reflection from the object. As used herein, directreflections and rebounded reflections refer to the various descriptionsprovided herein as well as the generally understood and variations ofthese terms.

A mapping system module 584 generates direct positional data associatedwith the object based on one or more direct ultrasonic reflections. Themapping system module 584 may also generate direct positional dataassociated with the object based on one or more indirect ultrasonicreflections, as may be understood in the art. The mapping system module584 may also generate rebounded positional data associated with theobject based on one or more indirect ultrasonic reflections, as may beunderstood in the art.

A direct reflection module 586 may be configured to facilitate, manage,and/or monitor the transmission and/or reception of direct reflections.The rebounded reflection module 588 may be configured to facilitate,manage, and/or monitor the transmission and/or reception of reboundedreflections.

The positional data calculation module 589 may generate directpositional data associated with the object based on one or more directultrasonic reflections. The positional data calculation module 589 mayalso generate rebounded positional data associated with the object basedon one or more rebounded ultrasonic reflections. The positional datacalculation module 589 may also generate enhanced positional data bycombining the direct positional data and the rebounded positional data.

FIG. 6 illustrates a flow chart of method 600 for generating positionaldata describing a relative position and/or movement of one or moreobjects within a region. The method steps are provided in no particularorder and may be rearranged as would be technically feasible. Apositioning system may transmit 605 ultrasound into a region bounded byat least one surface. The positioning system may receive 610 directultrasonic reflections from at least one object within the region.

The positioning system may receive 612 rebounded ultrasonic reflectionsfrom at least one object within the region. The rebounded ultrasonicreflections may reflect off the wall(s) first and/or off the object(s)first. The positioning system may generate 614 positional data based onthe direct reflections from the object. The positioning system maygenerate 616 positional data based on the rebounded reflections from theobject.

The positioning system may generate 618 enhanced positional data bycombining the direct positional data and the rebounded positional data.In other embodiments, the positioning system may transmit the directpositional data and the rebounded positional data to another electronicor other processing device for usage.

Any of the various configurations of ultrasonic transmitters, receivers,reflectors, and/or other components described in conjunction with thedetection of the position of an object may also be applied to theembodiments described herein with respect to the detection and/orcalculation of velocity and/or acceleration data associated with anobject or objects, including those embodiments described below withreference to FIGS. 7A-12. For example, direct and rebounded reflections,multiple reflectors and/or ultrasonic paths may be used to calculatevelocity and/or acceleration data associated with an object within aregion.

FIG. 7A illustrates an ultrasonic system 710, which can be used as partof a dual modality system, transmitting 720 and receiving 740 reflectedultrasound from a stationary object 730. The spacing between the arcsrepresenting the ultrasound 720 and 740 is representative of thewavelength and/or frequency of the ultrasound. With the object 730 in astationary position, the reflected ultrasound 740 is not shifted withrespect to the transmitted ultrasound 720.

FIG. 7B illustrates the ultrasonic system 710 transmitting ultrasound720 at a first frequency and receiving reflected ultrasound 741 at asecond frequency from an object moving away from the ultrasound system710. The frequency shift can be detected and used to determine thevelocity of the reverse motion of the object 730. For example, thevelocity of the object 730, V_(o), is equal to the change in frequency,Δf, multiplied by the velocity of the ultrasound, V_(us), divided by thefrequency of the transmitted ultrasound, f_(trans), relative to theultrasonic receiver. Any of a wide variety Doppler shift velocity and/oracceleration calculation and/or estimation algorithms may be utilized.

FIG. 7C illustrates an ultrasound system 710 transmitting ultrasound 720at a first frequency and receiving reflected ultrasound 742 at a secondfrequency from an object 730 moving toward the ultrasound system 710.Again, any of a wide variety of Doppler shift algorithms forcalculating, determining, and/or estimating the relative velocity of theobject 730 with respect to the ultrasonic system 710 may be used. Forexample, the Doppler equation:

$\begin{matrix}{f_{r} = {\left( \frac{C + V_{r}}{C + V_{o}} \right)f_{t}}} & {{Equation}\mspace{14mu} 1}\end{matrix}$

In equation 1 above it is assumed that a transmission medium (e.g., air)is relatively stationary, f_(r) is the frequency of the receivedultrasound, C is the velocity of the ultrasound in the medium (e.g.,air), V_(r) is the velocity of the ultrasonic receiver relative to themedium, V_(o) is the velocity of the object relative to the medium, andf_(t) is the frequency of the transmitted ultrasound. An acceleration ofthe object may be determined using velocity calculations at multiplediscrete time periods and/or by detecting a change in the frequency ofthe received ultrasound, f_(r), over time.

As described herein, the ultrasonic system 710 may include one or moreultrasonic transmitters and/or ultrasonic receivers and the transmittersand receivers may be physically joined (as illustrated in FIG. 7C) orthey may be separated and even possible positioned in disparatelocations within the region. In some embodiments, the transmitters andreceivers may be embodied in a single transducer. In other embodiments,each transducer may act as both an ultrasound transmitter and anultrasound receiver. In yet other embodiments each transducer may beused in conjunction with an electromagnetic position detection systemand/or receiver system as a dual modality system.

FIG. 7D illustrates an ultrasonic system 710 transmitting and receivingreflected ultrasound 720 and 743 from a stationary object 730, similarto FIG. 7A. FIG. 7D provides a representative context for FIG. 7E.

FIG. 7E illustrates a timing delay and/or phase shift, illustrated asmissing wave arc 757, in reflected ultrasound 743 from the object 730 asit moves away from the ultrasound system 710. As provided herein Dopplershifts may be used to determine acceleration and/or velocity informationassociated with a moving object. It is, however, recognized the variousmethods of velocity measurement may be utilized. Including, for example,phase shift (i.e., when received signals arrive) measurements, similarto those used in Doppler echocardiography. It is appreciated thatvarious 1D, 2D, and 3D vector Doppler calculations of velocity and/oracceleration information of an object may be incorporated into thepresently described systems and methods, including, but not limited to,2D Doppler Imaging, Vector Doppler, Speckle Tracking, and others.

FIG. 8 illustrates ultrasound 820 rebounded, at 821, off of a reflector850 (e.g., an auxiliary reflector) prior to being reflected by an object830 moving away from an ultrasound receiver 810. A shift in the receivedultrasound 840 relative to the transmitted ultrasound 820 can be used todetermine a velocity of the object 830 relative to the ultrasonicreceiver 810.

In one embodiment, the ultrasound may first be reflected by the object830, and then rebounded by the reflector 850. In such an embodiment, itmay be possible to determine velocity and/or acceleration information ofthe object 830 relative to the reflector 850.

FIG. 9 Illustrates an electromagnetic position detection system 913 usedin conjunction with an ultrasound velocity and/or acceleration detectionsystem 910 as a dual modality system. The ultrasonic velocity and/oracceleration detection system 910 may operate and/or be configured inconjunction with any of the various embodiments described herein fordetermining position, velocity, and/or acceleration information at acurrent time and/or for estimating such information at a future time.The electromagnetic position detection system 913 may detect a positionof the object 930 using time-of-flight. For example, a laser or otherelectromagnetic radiation source may be used to measure a time-of-flightbetween the system 913 and the object 930. The position informationobtained via an electromagnetic system 913 may be used in conjunctionwith velocity and/or acceleration data obtained using the ultrasonicsystem 910 described herein.

FIG. 10 illustrates ultrasound 1020 reflected and/or rebounded from oneor more auxiliary reflectors 1030 and 1040. As described in variousembodiments, an ultrasound receiver/transmitter 1010 may utilize directreflections from an object within a region to determine velocity and/oracceleration information based on a detected frequency shift and/orphase shift. In some embodiments, rebounded ultrasonic reflections maybe utilized in addition to or instead of direct ultrasonic reflections.Ultrasound reflectors 1030 and 1040 may be active or passive and may beintegrated into one or more appliances, walls, or other features of theregion. In some embodiments, existing walls, room features, furniture,people, objects, or the like may be identified and/or specified asreflectors 1030 and 1040.

FIG. 11 illustrates a plurality of ultrasonic systems 1120, 1121, 1122,and 1123 for determining velocity and/or acceleration information frommultiple directions relative to the object 1110 or a site on object 1110within a region 1100. In various embodiments, each ultrasonic system1120-1123 may include one or more ultrasonic transmitters and one ormore ultrasonic receivers. In other embodiments, one or more of theultrasonic systems 1120-1123 may include one or more ultrasonictransmitters or one or more ultrasonic receivers. In some embodiments,the ultrasonic transmitters and ultrasonic receivers may be separatecomponents spaced apart from one another. As illustrated, the ultrasoundmay be rebounded off of one or more auxiliary reflectors 141, 142, 143,and 144.

FIG. 12 illustrates a method 1200 for determining velocity and/oracceleration information associated with a moving object. Ultrasound maybe transmitted 1205 into a region bounded by at least one surface. Someembodiments may utilize direct reflections from the object to determinevelocity and/or acceleration data based on a detected shift in theultrasound, as provided in block 1240. A receiver may receive 1210direct ultrasound reflections from at least one object or a site on anobject within the region. A shift, such as a wavelength shift, frequencyshift, or phase shift, may be determined 1211 between the transmittedultrasound and the received ultrasound. The system may then generate1214 velocity and/or acceleration data based on the detected shift.

It is understood that “determining a shift,” “detecting a shift,”“calculating a shift,” and the like may not necessarily require anactual determination of the difference between the, e.g., frequency, ofthe transmitted and received ultrasound. That is, “detecting a shift”and similar phrases may be constructively performed during a Dopplercalculation of velocity and/or acceleration. For example, “detecting ashift” may be constructively performed if a velocity of an object isdetermined using (1) a known/measured frequency of transmittedultrasound and (2) a known/measured frequency of ultrasound reflected bythe object. The system may or may not actually calculate the frequencydifference between the transmitted and received ultrasound, as variousderivative and equal algorithms for Doppler-based velocity calculationsmay be utilized.

In some embodiments, rebounded reflections from the object may be usedto determine velocity and/or acceleration data based on a detected shiftin the ultrasound, as provided in block 1250. Ultrasound may betransmitted 1205 into a region bounded by at least one surface. Areceiver may receive 1212 rebounded ultrasound reflections from at leastone object or a site on an object within the region. A shift, such as awavelength shift, frequency shift, or phase shift, may be determined1213 between the transmitted ultrasound and the received ultrasound. Thesystem may then generate 1216 velocity and/or acceleration data based onthe detected shift. In various embodiments, velocity and/or accelerationdata from direct reflections and rebounded reflections may be optionallycombined 1218. Velocity and/or acceleration data from direct reflectionsand rebounded reflections may be used to determine two-dimensionalvectors of velocity and/or acceleration information related to theobject or a site on the object.

FIG. 13A illustrates a dual modality system 1310 a transmittingultrasound 1325 a and receiving ultrasound 1327 reflected off a person1362 a standing within a bounded region 1300 a. As illustrated, thebounded region 1300 a is bounded by a floor 1341 a, a left wall 1342 a,a back wall 1343 a, a right wall 1344 a, and a ceiling 1345 a. A frontwall (not shown), may also bound the region 1300 a.

The dual modality system 1310 a may utilize its ultrasound modalitysimilar to that discussed in relation to the previous figures. Forexample, the dual modality system 1310 a may transmit the ultrasound1325 a as directional ultrasonic pulses, continuously, in a modulatedfashion (frequency, amplitude, phase, etc.), and/or in another format.The ultrasound 1325 a may be transmitted directly toward the person 1362a and/or the ultrasound 1325 a may be transmitted indirectly toward theperson 1362 a.

FIG. 13B illustrates a dual modality system 1310 b generating coarseimage data of an object based on received ultrasonic reflections. Asillustrated, the direct ultrasonic reflection 1327 b may convey coarseimage information in a relatively two-dimensional fashion. Dual modalitysystem 1310 b may use a processor to generate coarse image data of aperson 1362 b or other object. The actual image data received may be ata higher or lower resolution depending on the sampling rates, accuracy,processing bit depth, frequency(ies) of ultrasound used, etc. In certainexamples, to use less processing power, the dual modality system 1310 bmay use a low resolution to track the object as a whole. If a lowresolution is used, portions of interest may be identified for whichmore detailed information is desirable, useful, and/or necessary for aparticular application.

FIG. 13C illustrates a dual modality system 1310 c identifying severalportions of interest on an object using on coarse image data generatedfrom ultrasonic reflections 1327 c (or alternatively using receivedelectromagnetic radiation). Portions of interest on a person 1362 c mayinclude a hand, finger, arm, leg, foot, toe, torso, neck, head, mouth,lip, and eye. For example, as illustrated in FIG. 13C dual modalitysystem 1310 c has identified the hands, feet, and head as portions ofinterest on person 1362 c.

The portion of interest may be identified based partly on the state ofan associated entertainment device. For example, a state of theassociated entertainment device may utilize a hand movement for aparticular action that cannot be determined using coarse image data. Inthat situation, the dual modality system 1310 c may identify the handsof a person 1362 c as a portion of interest for which fine image data(i.e., higher resolution images) are desired. Whatever portions ofinterest are identified, the dual modality system 1310 c may use asecond modality to receive additional and more detailed imageinformation.

For example, FIG. 14 illustrates a dual modality system 1410 usingelectromagnetic imaging in conjunction with ultrasound 1425 to receiveadditional image information from identified portion of interests. Inaddition to ultrasound, the dual modality system 1410 may useelectromagnetic transmitters and/or receivers to receive/determineelectromagnetic image information. For example, a laser or otherelectromagnetic radiation source may be used to receive image data of anobject. In certain examples, the dual modality system 1410 may use areceiver such as an infrared receiver to gather image data from ambientradiation sources. In certain examples, electromagnetic radiationreceived may include microwave, terahertz, infrared, visible, and/orultraviolet radiation.

A system may use electromagnetic imaging capabilities to either receivecoarse image data at a low-resolution or fine image data at a higher,more detailed resolution. For example, FIG. 14 illustrates a dualmodality system 1410 receiving electromagnetic radiation from anidentified portion of interest on a two-dimensional coarse imagegenerated using received ultrasound reflections 1427. Theelectromagnetic radiation received may be used to generate a higherresolution image than the image generated using the received ultrasoundreflections 1427.

For example, FIG. 15 illustrates a dual modality system 1510 generatingfine image data of portions of interest on a person 1562. One modality,such as ultrasound 1525, may be used to generate a coarse image with alow resolution while a second modality, such as electromagneticradiation, may be used to generate a detailed image with a higherresolution. It will be understood that either ultrasound orelectromagnetic radiation may be utilized to gather the low-resolutionimage or the higher resolution image.

FIG. 16 illustrates a method for generating an image using ultrasoundand electromagnetic radiation. The method steps are provided in noparticular order and may be rearranged as would be technically feasible.A dual modality system may transmit 1605 ultrasound into a regionbounded by at least one surface. The dual modality system may receive1607 direct or rebounded ultrasonic reflections from at least one objectwithin the region. Using the received ultrasonic reflection, the dualmodality system may generate 1609 coarse image data of the object. Thedual modality system may identify 1611 one or more portions of intereston an object. The dual modality system may receive 1613 electromagneticradiation from the portion(s) of interest, and using the electromagneticradiation, generate 1615 fine image data of the object. The fine imagedata may be of a higher resolution than the coarse image data.Optionally, the system may also determine 1617 a kinematic valueassociated with the portion of interest, and modify 1619 the state of anentertainment device based on the kinematic value.

FIG. 17 illustrates a method for generating an image using ultrasoundand electromagnetic radiation. The method steps are provided in noparticular order and may be rearranged as would be technically feasible.The dual modality system may receive 1702 electromagnetic radiation froma plurality of sites within a region. In some embodiments, the receivedelectromagnetic radiation may be used to generate a coarse image of anobject or site on an object with the region. The dual modality systemmay identify 1704 one or more portions of interest on an object. Thedual modality system may transmit 1706 ultrasound into the region, andreceive 1708 direct or rebounded ultrasonic reflections from the portionof interest. From the received ultrasonic reflection, the dual modalitysystem may generate 1710 fine (higher resolution) image data of theobject and/or portions of interest on the object. Optionally, the systemmay also determine 1712 a kinematic value associated with the portion ofinterest, and modify 1714 the state of an entertainment device based onthe kinematic value.

FIG. 18 illustrates a method for resolving ambiguities in an image usingultrasound and electromagnetic radiation. The method steps are providedin no particular order and may be rearranged as would be technicallyfeasible. A dual modality system may transmit 1805 ultrasound into aregion bounded by at least one surface. The dual modality system mayreceive 1807 direct or rebounded ultrasonic reflections from at leastone object within the region. Using the received ultrasonic reflection,the dual modality system may generate 1809 image data of the object. Theimage data generated using the received ultrasonic reflection mayinclude at least one ambiguity, such as a ghost image. The dual modalitysystem may receive 1811 electromagnetic radiation from the object thatis sufficient to resolve the ambiguity. The dual modality system maygenerate 1813 enhanced image data that resolves the ambiguity based onthe received electromagnetic radiation. Optionally, the system may alsodetermine 1815 a kinematic value associated with the portion of interestand/or modify 1817 the state of an entertainment device based on thekinematic value.

This disclosure has been made with reference to various exemplaryembodiments, including the best mode. However, those skilled in the artwill recognize that changes and modifications may be made to theexemplary embodiments without departing from the scope of the presentdisclosure. While the principles of this disclosure have been shown invarious embodiments, many modifications of structure, arrangements,proportions, elements, materials, and components may be adapted for aspecific environment and/or operating requirements without departingfrom the principles and scope of this disclosure. These and otherchanges or modifications are intended to be included within the scope ofthe present disclosure.

This disclosure is to be regarded in an illustrative rather than arestrictive sense, and all such modifications are intended to beincluded within the scope thereof. Likewise, benefits, other advantages,and solutions to problems have been described above with regard tovarious embodiments. However, benefits, advantages, solutions toproblems, and any element(s) that may cause any benefit, advantage, orsolution to occur or become more pronounced are not to be construed as acritical, required, or essential feature or element. The scope of thepresent invention should, therefore, be determined by the followingclaims:

What is claimed is:
 1. A method for generating an image of an objectwithin a region, comprising: transmitting ultrasound, via a firstultrasonic transmitter of an ultrasonic transmission system, into theregion; receiving, via a first ultrasonic receiver of an ultrasonicreceiver system, ultrasonic reflections of the transmitted ultrasoundfrom a plurality of sites on the object within the region, theultrasonic reflections comprising sound waves within a first frequencyband, the ultrasonic reflections including direct ultrasonic reflectionand rebounded ultrasonic reflection, wherein the direct ultrasonicreflection is reflected from a first portion of the object and therebounded ultrasonic reflection is reflected from a second, differentportion of the object; generating, via a processor, coarse image data ofthe object at a first resolution based on the received ultrasonicreflections; identifying a portion of interest on the object based onthe coarse image data; receiving electromagnetic radiation from theidentified portion of interest on the object, the electromagneticradiation comprising electromagnetic waves within a second frequencyband, the second frequency band being different than the first frequencyband; generating fine image data of the portion of interest on theobject at a second resolution based on the received electromagneticradiation, wherein the second resolution is greater than the firstresolution; determining a kinematic value associated with the portion ofinterest on the object based on at least one of the receivedelectromagnetic radiation and the received ultrasonic reflections,wherein positional data is generated based at least on the directultrasonic reflection and the rebounded ultrasonic reflection, whereinthe kinematic value is determined based at least in part on thepositional data; and modifying the state of an entertainment devicebased on the determined kinematic value associated with the portion ofinterest on the object.
 2. The method of claim 1, wherein the portion ofinterest is identified based at least in part on a state of theentertainment device.
 3. The method of claim 1, further comprisingtransmitting electromagnetic energy into the region via anelectromagnetic transmitter, and wherein receiving electromagneticradiation from the identified portion of interest on the objectcomprises receiving a reflected portion of the transmittedelectromagnetic radiation.
 4. The method of claim 1, wherein at leastone of the transmitted ultrasound and the received reflected ultrasoundare rebounded from an auxiliary ultrasonic reflector.
 5. The method ofclaim 1, wherein calculating the kinematic value associated with theportion of interest on the object comprises a Doppler velocitycalculation in which the first velocity (V_(dop)) is a function of: acalculated shift (Δ_(s)), the frequency of the transmitted ultrasound(f₀), an arrival velocity of the ultrasound (V_(in)), and a departurevelocity of the ultrasound (V_(out)).
 6. The method of claim 1, whereindetecting a shift and calculating the first velocity component areperformed using a derivation or equivalent equation to the followingequation: ${f_{r} = {\left( \frac{C - V_{o}}{C + V_{o}} \right)f_{t}}},$where f_(r) is based on the frequency of the received ultrasound, C isbased on a velocity of the ultrasound in a medium within the region,V_(o) is based on a velocity of the site on the object relative to themedium, and f_(t) is based on a frequency of the transmitted ultrasound.7. The method of claim 1, further comprising: receiving, via a secondultrasonic receiver of the ultrasonic receiver system that is physicallyseparated from the first ultrasonic receiver, an ultrasonic reflectionfrom the object; detecting a shift of the ultrasonic reflection receivedby the second ultrasonic receiver; and calculating, via the processor, asecond kinematic value associated with the portion of interest on theobject based on the detected shift of the ultrasonic reflection receivedby the second ultrasonic receiver.
 8. The method of claim 7, furthercomprising: receiving, via a third ultrasonic receiver of the ultrasonicreceiver system that is physically separated from the first ultrasonicreceiver, an ultrasonic reflection from the object; detecting a shift ofthe ultrasonic reflection received by the third ultrasonic receiver; andcalculating, via the processor, a third kinematic value associated withthe portion of interest on the object based on the detected shift of theultrasonic reflection received by the third ultrasonic receiver.
 9. Themethod of claim 8, wherein the first, second and third ultrasonicreceivers are non-collinear with respect to one another.
 10. The methodof claim 1, wherein a plurality of kinematic values are calculated for acorresponding plurality of sites on the object.
 11. The method of claim10, wherein the plurality of kinematic values are used to determine atranslational velocity of the object.
 12. The method of claim 1, whereinthe portion of interest on the object comprises a portion of a personselected from the group consisting of a hand, finger, arm, leg, foot,toe, torso, neck, head, mouth, lip, and eye.
 13. The method of claim 1,wherein determining the kinematic value of the portion of interest onthe object comprises: transmitting ultrasound, via the ultrasoundtransmission system, into the region, wherein the region is bounded by afirst surface; receiving, via the ultrasonic receiver system, the directultrasonic reflection from the object; generating direct positional dataassociated with the object based on the direct ultrasonic reflection;receiving the rebounded ultrasonic reflection from the object, whereinthe rebounded ultrasonic reflection comprises ultrasound reflected bythe object and the first surface prior to being received by theultrasonic receiver; generating rebounded positional data using therebounded ultrasonic reflection of the object from the first surface;and generating enhanced positional data by combining the directpositional data and the rebounded positional data.
 14. The method ofclaim 1, wherein determining the kinematic value of the portion ofinterest on the object comprises: transmitting ultrasound into theregion, wherein the region is bounded by a first surface; receiving thedirect ultrasonic reflection from the object; generating directpositional data associated with the object based on the directultrasonic reflection; receiving a rebounded ultrasonic reflection fromthe object, wherein the rebounded ultrasonic reflection comprisesultrasound reflected by the object and the first surface prior to beingreceived; generating rebounded positional data using the reboundedultrasonic reflection of the object from the first surface; andgenerating enhanced positional data by combining the direct positionaldata and the rebounded positional data.
 15. The method of claim 14,further comprising: receiving an additional rebounded ultrasonicreflection from the object, wherein the additional rebounded ultrasonicreflection comprises ultrasound reflected by the object and a secondsurface bounding the region prior to being received; generatingadditional rebounded positional data using the additional reboundedultrasonic reflection of the object from the second surface; andsupplementing the enhanced positional data with the additional reboundedpositional data.
 16. The method of claim 14, wherein transmitting theultrasound comprises: transmitting a first ultrasonic pulse that isreceived as the direct ultrasonic reflection; and transmitting a secondultrasonic pulse that is received as the rebounded ultrasonicreflection.
 17. A method for generating an image of an object within aregion, comprising: receiving electromagnetic radiation from a pluralityof sites on an object within a region, the electromagnetic radiationcomprising electromagnetic waves within a first frequency band;identifying a portion of interest on the object based on the receivedelectromagnetic radiation; transmitting ultrasound, via a firstultrasonic transmitter of an ultrasonic transmission system, into theregion; receiving, via a first ultrasonic receiver of an ultrasonicreceiver system, ultrasonic reflections of the transmitted ultrasoundfrom the portion of interest on the object within the region, theultrasonic reflections comprising sound waves within a second frequencyband, the second frequency band being different than the first frequencyband, the ultrasonic reflections including direct ultrasonic reflectionand rebounded ultrasonic reflection, wherein the direct ultrasonicreflection is reflected from a first portion of the portion of intereston the object and the rebounded ultrasonic reflection is reflected froma second, different portion of the portion of interest on the object;generating, via a processor, image data of the portion of interest onthe object at a first resolution based on the received ultrasonicreflections; determining a kinematic value associated with the portionof interest based on at least one of the received electromagneticradiation and the received ultrasonic reflections, wherein positionaldata is generated based at least on the direct ultrasonic reflection andthe rebounded ultrasonic reflection, wherein the kinematic value isdetermined based at least in part on the positional data; and modifyingthe state of an entertainment device based on the determined kinematicvalue associated with the portion of interest on the object.
 18. Themethod of claim 17, further comprising generating image data of theobject at a second resolution based on the received electromagneticradiation, wherein the second resolution is less than the firstresolution.
 19. The method of claim 17, further comprising transmittingelectromagnetic radiation into the region.
 20. A method for generatingan image of an object within a region, comprising: transmittingultrasound, via an ultrasonic transmission system, into the region;receiving, via an ultrasonic receiver system, ultrasonic reflections ofthe transmitted ultrasound from a plurality of sites on the objectwithin the region, the ultrasonic reflections comprising sound waveswithin a first frequency band, the ultrasonic reflections includingdirect ultrasonic reflection and rebounded ultrasonic reflection,wherein the direct ultrasonic reflection is reflected from a firstportion of the object and the rebounded ultrasonic reflection isreflected from a second, different portion of the object; generating,via a processor, image data of the object at a first resolution based onthe received ultrasonic reflections, wherein the image data at the firstresolution includes at least one ambiguity; receiving electromagneticradiation reflected by the object that is sufficient to resolve the atleast one ambiguity, the electromagnetic radiation comprisingelectromagnetic waves within a second frequency band, the secondfrequency band being different than the first frequency band; generatingenhanced image data resolving the at least one ambiguity based on thereceived electromagnetic radiation; determining a kinematic valueassociated with the object based on at least one of the receivedelectromagnetic radiation and the received ultrasonic reflections,wherein positional data is generated based at least on the directultrasonic reflection and the rebounded ultrasonic reflection, whereinthe kinematic value is determined based at least in part on thepositional data; and modifying the state of an entertainment devicebased on the determined kinematic value associated with the portion ofinterest on the object.
 21. The method of claim 20, wherein resolvingthe at least one ambiguity comprises determining which of a plurality ofimages associated with the image data is a ghost image.
 22. The methodof claim 20, further comprising directing electromagnetic radiationtowards a site on the object associated with the at least one ambiguity.23. A system for generating an image of an object within a region,comprising: an ultrasonic transmitter configured to transmit ultrasoundinto the region; an ultrasonic receiver configured to receive anultrasonic reflection of the transmitted ultrasound from a plurality ofsites on the object within the region, the ultrasonic reflectionscomprising sound waves within a first frequency band, the ultrasonicreflection including a direct ultrasonic reflection and a reboundedultrasonic reflection, wherein the direct ultrasonic reflection isreflected from a first portion of the object and the reboundedultrasonic reflection is reflected from a second, different portion ofthe object; a first imaging module configured to generate coarse imagedata of the object at a first resolution based on the receivedultrasonic reflections; an identification module configured to identifya portion of interest on the object based on the coarse image data; anelectromagnetic radiation receiver configured to receive electromagneticradiation from the identified portion of interest on the object, theelectromagnetic radiation comprising electromagnetic waves within asecond frequency band, the second frequency band being different thanthe first frequency band; a second imaging module configured to generatefine image data of the portion of interest on the object at a secondresolution based on electromagnetic radiation received by theelectromagnetic radiation receiver, wherein the second resolution isgreater than the first resolution; a kinematic determination moduleconfigured to determine a kinematic value associated with the portion ofinterest on the object based on at least one of received electromagneticradiation and received ultrasonic reflections, wherein positional datais generated based at least on the direct ultrasonic reflection and therebounded ultrasonic reflection, wherein the kinematic value isdetermined based at least in part on the positional data; and amodification module configured to modify the state of an entertainmentdevice based on the kinematic value associated with the portion ofinterest on the object.
 24. The system of claim 23, wherein thekinematic value of the object comprises a velocity of the object. 25.The system of claim 23, wherein at least one of the transmittedultrasound and the reflected ultrasound are rebounded from an auxiliaryultrasonic reflector.
 26. The system of claim 23, further comprising: asecond ultrasonic receiver that is physically separated from the firstultrasonic receiver, wherein the second ultrasonic receiver isconfigured to receive an ultrasonic reflection from the site; wherein ashift module is configured to detect a shift of the ultrasonicreflection received by the second ultrasonic receiver, and wherein thekinematic value module is configured to calculate a second kinematicvalue associated with the portion of interest on the object based on thedetected shift of the ultrasonic reflection received by the secondultrasonic receiver.
 27. The system of claim 26, further comprising: athird ultrasonic receiver that is physically separated from the firstand second ultrasonic receivers, wherein the third ultrasonic receiveris configured to receive an ultrasonic reflection from the site, whereinthe shift module is configured to detect a shift of the ultrasonicreflection received by the third ultrasonic receiver, and wherein thekinematic value module is configured to calculate a third kinematicvalue associated with the portion of interest on the object based on thedetected shift of the ultrasonic reflection received by the thirdultrasonic receiver.
 28. The system of claim 27, further comprising: aprediction module configured to predict a relative position of theobject at a future time based on (1) the first kinematic valueassociated with the portion of interest on the object, (2) the secondkinematic value associated with the portion of interest on the object,and (3) the third kinematic value associated with the portion ofinterest on the object.
 29. A system for generating an image of anobject within a region, comprising: an electromagnetic radiationreceiver configured to receive electromagnetic radiation from aplurality of sites on an object within a region, the electromagneticradiation comprising electromagnetic waves within a first frequencyband; an identification module configured to identify a portion ofinterest on the object based on the received electromagnetic radiation;an ultrasonic transmitter configured to transmit ultrasound into theregion; an ultrasonic receiver configured to receive an ultrasonicreflection of the transmitted ultrasound from the portion of interest onthe object within the region, the ultrasonic reflections comprisingsound waves within a second frequency band, the second frequency bandbeing different than the first frequency band, the ultrasonic reflectionincluding a direct ultrasonic reflection and a rebounded ultrasonicreflection, wherein the direct ultrasonic reflection is reflected from afirst portion of the object and the rebounded ultrasonic reflection isreflected from a second, different portion of the object; a firstimaging module configured to generate image data of the portion ofinterest on the object at a first resolution based on the receivedultrasonic reflections; a kinematic determination module configured todetermine a kinematic value associated with the portion of interest onthe object based on at least one of received electromagnetic radiationand received ultrasonic reflections, wherein positional data isgenerated based at least on the direct ultrasonic reflection and therebounded ultrasonic reflection, wherein the kinematic value isdetermined based at least in part on the positional data; and amodification module configured to modify the state of an entertainmentdevice based on the kinematic value associated with the portion ofinterest on the object.
 30. The system of claim 29, wherein thekinematic value of the object comprises a position of the object. 31.The method of claim 1, wherein the first frequency band is between 20kilohertz (kHz) and 250 kHz.
 32. The method of claim 1, wherein thefirst frequency band is between 35 kHz and 45 kHz.
 33. The method ofclaim 17, wherein the second frequency band is between 20 kilohertz(kHz) and 250 kHz.
 34. The method of claim 18, wherein the secondfrequency band is between 35 kHz and 45 kHz.
 35. The method of claim 20,wherein the first frequency band is between 20 kilohertz (kHz) and 250kHz.
 36. The method of claim 20, wherein the first frequency band isbetween 35 kHz and 45 kHz.
 37. The system of claim 23, wherein the firstfrequency band is between 20 kilohertz (kHz) and 250 kHz.
 38. The systemof claim 23, wherein the first frequency band is between 35 kHz and 45kHz.
 39. The system of claim 29, wherein the second frequency band isbetween 20 kilohertz (kHz) and 250 kHz.
 40. The system of claim 29,wherein the second frequency band is between 35 kHz and 45 kHz.